Robust Minimax Probability Machine Regression ; CU-CS-952-03

We formulate regression as maximizing the minimum probability (Ω) that the true regression function is within ±2 of the regression model. Our framework starts by posing regression as a binary classification problem, such that a solution to this single classification problem directly solves the original regression problem. Minimax probability machine classification (Lanckriet et al., 2002a) is used to solve the binary classification problem, resulting in a direct bound on the minimum probability Ω that the true regression function is within ±2 of the regression model. This minimax probability machine regression (MPMR) model assumes only that the mean and covariance matrix of the distribution that generated the regression data are known; no further assumptions on conditional distributions are required. Theory is formulated for determining when estimates of mean and covariance are accurate, thus implying robust estimates of the probability bound Ω. Conditions under which the MPMR regression surface is identical to a standard least squares regression surface are given, allowing direct generalization bounds to be easily calculated for any least squares regression model (which was previously possible only under very specific, often unrealistic, distributional assumptions). We further generalize these theoretical bounds to any superposition of basis functions regression model. Experimental evidence is given supporting these theoretical results.